Focal length validation using three-dimensional pose estimates

ABSTRACT

A computer-implemented method includes receiving, by a computing system, a first image of an optical target from a camera. The first image is associated with a first position of the camera. The method also includes receiving, by the computing system, a second image of the optical target, Wherein the second image is associated with a second position of the camera. The method also includes determining an estimated displacement between the first position of the camera and the second position of the camera based on the first image and the second image. The method further includes determining a focal length based on the estimated displacement and a measured displacement of the camera between the first position and the second position.

CROSS-REFERENCE TO RELATED PATENT APPLICATION

This patent application is a continuation of U.S. patent applicationSer. No. 17/523,152, entitled “FOCAL LENGTH VALIDATION USINGTHREE-DIMENSIONAL POSE ESTIMATES,” by Yongjun Wang et al, filed on Nov.10, 2021, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

The subject technology provides methods for performing camera focallength validation. In particular, the disclosure provides methods thatcombine image-based pose estimations and Light Detection and Ranging(LiDAR) measurements for focal length validations.

BACKGROUND

An autonomous vehicle is a motorized vehicle that can navigate without ahuman driver. An exemplary autonomous vehicle includes a plurality ofsensor systems, such as, but not limited to, a camera sensor system, alidar sensor system, a radar sensor system, amongst others, wherein theautonomous vehicle operates based upon sensor signals output by thesensor systems. Specifically, the sensor signals are provided to aninternal computing system in communication with the plurality of sensorsystems, wherein a processor executes instructions based upon the sensorsignals to control a mechanical system of the autonomous vehicle, suchas a vehicle propulsion system, a braking system, or a steering system.In some applications, these systems utilize a perception system (orperception stack), that implements various computing vision techniquesin order to reason about the surrounding environment. In someapplications, three-dimensional (3D) pose estimation techniques may beused. As discussed in further detail below, a (3D) pose estimation is acomputer vision technique that predicts the transformation of an objectfrom a pose or an image and tracks the location of a person or object.The pose estimation is done by using a combination of the pose and theorientation of the person or object.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-recited and other advantages and features of the presenttechnology will become apparent by reference to specific implementationsillustrated in the appended drawings. A person of ordinary skill in theart will understand that these drawings only show some examples of thepresent technology and would not limit the scope of the presenttechnology to these examples. Furthermore, the skilled artisan willappreciate the principles of the present technology as described andexplained with additional specificity and detail through the use of theaccompanying drawings in which:

FIG. 1 is a diagram illustrating a method for comparing estimateddisplacements with measured displacements, according to some aspects ofthe disclosed technology;

FIG. 2 is a diagram illustrating an example system for implementing themethod of FIG. 1 , according to some aspects of the disclosedtechnology;

FIG. 3A illustrates example transformations of the camera of FIG. 2 atfive different positions with respect to the linear stage of FIG. 2 ,according to some aspects of the disclosed technology;

FIG. 3B illustrates an example optical target including a checkerboardpattern, according to some aspects of the disclosed technology;

FIG. 3C illustrates example displacements of the camera of FIG. 2 withrespect to the linear stage of FIG. 2 , according to some aspects of thedisclosed technology;

FIG. 4 illustrates camera coordinates X′, Y′, Z′ with an origin locatedat a camera center and define a focal length from an image plane,according to some aspects of the disclosed technology;

FIG. 5 illustrates errors on the camera plane of FIG. 4 , according tosome aspects of the disclosed technology;

FIG. 6A illustrates an example of image distortion, according to someaspects of the disclosed technology;

FIG. 6B illustrates an example of an undistorted image, according tosome aspects of the disclosed technology;

FIG. 6C illustrates another example of image distortion, according tosome aspects of the disclosed technology;

FIG. 7 illustrates an example method for performing focal lengthvalidation, according to some aspects of the disclosed technology;

FIG. 8 illustrates displacements from pose estimations with images andmeasured displacements from a LiDAR sensor or a Leica scanner, accordingto some aspects of the disclosed technology;

FIG. 9 illustrates simulated results of target distances versus focallengths, according to some aspects of the disclosed technology;

FIG. 10 illustrates an example environment that includes an autonomousvehicle in communication with a remote computing system, according tosome aspects of the disclosed technology; and

FIG. 11 shows an example of a computing system for implementing certainaspects of the present technology.

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detailbelow. While specific implementations are discussed, it should beunderstood that this is done for illustration purposes only. A personskilled in the relevant art will recognize that other components andconfigurations may be used without parting from the spirit and scope ofthe present technology. In some instances, well-known structures anddevices are shown in block diagram form to facilitate describing one ormore aspects. Further, it is to be understood that functionality that isdescribed as being carried out by certain system components may beperformed by more or fewer components than shown.

Validating the precise focal length of optical imaging devices, such ascameras, can be difficult due to the fact that image acquisition isperformed behind the lens at a location that is typically inside thedevice (e.g. camera) housing, making it difficult to precisely measurethe location of the lens. However, knowing precise optical parametersfor imaging devices is crucial for high-precision applications, such asthose used in autonomous vehicles (AVs) that rely on high-accuracylocalizations to better reason about the surrounding environment.Aspects of the disclosed technology provide solutions for ascertainingaccurate imaging parameters/characteristics, such as focal lengthparameters, for lens-based imaging devices, such as cameras.

In some approaches, focal lengths can be determined/validated byanalyzing and comparing various measures of device displacement. By wayof example, a focal length parameter of an imaging device, such as acamera, can be determined by estimating the device displacement betweenlocations where image collection is performed, for example, wherebyimages of a stationary target are acquired by a camera. By comparing theestimated displacement between locations with an accurate (groundtruth), measure of the device displacement, the focal length parametercan be accurately determined/validated. As discussed in further detailbelow, ground-truth measures of location displacement can be achievedusing various measurement means, such as by using a light-based rangingdevice, such as a LiDAR sensor or a laser ranging device. However, it isunderstood that such location/distance measurements may be performedusing other measurement device, without departing from the scope of thedisclosed technology.

The focal lengths can be validated with predicted positions and measuredpositions of 3D points. The positions of 3D points can be establishedwith both image-based pose estimations and those measured by LiDARsensors. However, these two methods for position estimations ormeasurements have some problems. For example, a LiDAR sensor can locatethe camera, but cannot determine accurately where the camera sensor isinside the camera. The image-based pose estimations can output positionsin camera-frames, but the image-based pose estimations are impacted bycalibration inaccuracy. Both the measurements from LiDAR sensors and theimage-based pose estimations are needed to construct 3D points incamera-frames accurately.

The accuracy of camera calibrations can be evaluated by computingdiscrepancies between expected positions based on the cameracalibrations and detected positions of known 3D points on an imageplane. It is critical to have a reliable and accurate way to acquire 3Ddata.

A traditional way for camera calibrations is to measure 3D points, thenproject the 3D points onto a two-dimensional (2D) image plane, thencalculate the errors, and estimate the accuracy. The traditional methodconstructs 3D feature points in camera-frames by measuring features andcamera lens/image sensor center using 2D or 3D equipment, which is proneto measurement errors as lens/sense centers and features on checkerboardare hard to locate accurately. The challenge is that it is difficult toknow the accurate location of the sensor inside the camera. The problemis solved by correcting estimated camera poses with precisely measuredtarget displacements, then use calibrated pose estimations to construct3D points in camera coordinates. This way does not need to locatefeatures or camera lens/image centers with instruments.

The disclosed method combines image-based pose estimations and accuratedistance measurements by LiDAR sensors or scanners to construct 3Dpoints reliably (e.g. 3D points of the initial position of the camerafrom which the camera moves to multiple stops or location of the cameracenter), which are further used to validate camera calibrations bycomparing predicted and detected image points.

The disclosed method calibrates image-based pose estimations with LiDARmeasurements by moving cameras between stops. Then, the movementsbetween stops measured with LiDAR sensors can be correlated with theimage-based pose estimations. A correction can then be applied toimage-based pose estimations in an initial position of the camera.

Target positions in all the stops in the camera frame can be constructedby combining the initial position with relative camera movements. Then,3D points (e.g. corners of checkers) are provided based upon targetdesigns (e.g. checkerboard). Those 3D points can be projected into 2Dpoints with camera calibrations and compare with detected or measuredimage points to evaluate calibration accuracy.

The disclosed method uses accurately measured distances to calibratecamera pose estimations, and further constructs 3D points directly incamera-frames without locating camera lens or image center or features.This avoids the use of other measurement equipment to locate features orcamera lens/image centers.

FIG. 1 is a diagram illustrating the method for comparing estimateddisplacements with measured displacements, according to some aspects ofthe disclosed technology. As illustrated in FIG. 1 , based uponcalibration parameters 102 and target images 104, pose estimation 108can be made. In some aspects, the calibration parameters 102 canrepresent a priori calibration estimates of the imaging device beingused to collect the target images 104, such as those that may beprovided by a camera manufacturer. The target images 104 can be taken bya camera moving to multiple stops. Then, estimated displacements 112(e.g., between stops) can be determined based upon the pose estimation108 at each stop. Also, separate measures of camera displacement betweenstops can be made using a high-accuracy measurement device, such asusing an optical (laser) based ranging device 106. Depending on thedesired implementation, ranging device 106 may include one or more LiDARsensors 106, and/or a laser-based ranging device, such as a scanner madeby Leica™.

The estimated displacements from the camera images can correlate withthe measured displacements. For example, if the images have nodistortion, the estimated displacements from the camera are linearlycorrelated with the measured displacements. A ratio (e.g. slope) can beobtained from the correlation between the estimated displacements andthe measured displacements. Then, a focal length can be corrected basedupon the ratio. More details will be illustrated in more details later.

FIG. 2 is a system diagram that illustrates an example of the method ofFIG. 1 , according to some aspects of the disclosed technology. Asillustrated in FIG. 2 , three targets 208A-C are positioned at twodifferent distances from a camera 202 at stops 210A and 210B on a linearstage 204. The targets 208A-C are placed at different angles from thelinear stage 204. For example, target 208A is at an angle of 90° orperpendicular to the linear stage 204. Target 208B is at a negativeangle from the linear stage 204 while target 208C is at a positive anglefrom the linear stage 204. A light-based ranging device 206 or a LiDARsensor is positioned at a distance from the targets 208A-C and also at adistance from the camera 202. The light-based ranging device 206 canmeasure the position of camera 202 at different stops 210A-B. Themeasurements by light-based ranging device 206 are from one cameraposition to another camera position light-based ranging device 206 doesnot perform measurements of the distance between the target and thecamera.

One may move camera 202 to multiple stops on the linear stage 204relative to target 208A to determine distances between stops with cameracalibration information. One may catch images of the target at each stopby the camera and estimate the displacements between the stops basedupon the images. One may also use a light-based ranging device to trackthe movements of the camera and to measure distances between the stops,and then correlates the estimated displacements with the measureddisplacements.

The disclosed method uses multiple targets and multiple stops for robustcalibrations between LiDAR measurements and image-based poseestimations. Mathematical calculations are provided below for estimatingdisplacements based upon image-based pose estimations.

In some aspects, the method may include correcting an image-basedposition estimate of the first target at the initial position,constructing 3D points in camera coordinates based upon the correctedimage-based position estimate at the initial position and measureddisplacements, and calibrating the camera.

FIG. 3A illustrates example transformations of the camera of FIG. 2 atfive different positions with a target at an angle less than 90 degreeswith respect to the linear stage of FIG. 2 , according to some aspectsof the disclosed technology.

As shown in FIG. 3A, camera 202 moves on the linear stage 204 from afirst stop 310A to four stops 310B-E sequentially. A target 308A ispositioned at a distance from stops 310A-B, which is referred to astransformation TE1, TE2, TE3, TE4, TE5 from stops 310A-E, respectively.Relative transformation TEi_rel between Nth stop (where N is an integerequal to or greater than 2) and the first stop can be calculated afterthe target positions (e.g. TE1-5) at each of stops 310A, 310B, 310C,310D, and 310E are extracted from images using Eq. (1) as follows:

TEi_rel=TE1_inverse*TEi   Eq. (1)

The first stop 310A can be used as a reference for LiDAR measurements,then relative transformation TLi between the first stop and another stop(e.g. one of stops 310B-E) can be extracted from the LiDAR measurements,where i is an integer equal to or greater than 1. Then, relativetransformations from the images and LiDAR measurements can be compared.

For example, TE1_rel is the relative transformation between the firststop 310A and the second stop 310B and is obtained from the images takenat the first stop 310A and the second stop 310B. TL1 is the relativetransformation measured by LiDAR, i.e. the displacement or distancebetween the first stop 310A and the second stop 310B. TE1_rel can becompared with TL1. Likewise, TE2_rel is the relative transformationbetween the first stop 310A and the third stop 310C and is obtained fromthe images taken at the first stop 310A and the third stop 310C. TL2 isthe relative transformation measured by LiDAR, i.e. the displacement ordistance between the first stop 310A and the third stop 310C. TE2_relcan be compared with TL2.

Then, corrections (also a transformation) can be computed forimage-based pose estimations using Eq. (2) as follows:

correction=TLi*TEi_rel_inverse   Eq. (2)

Corrections can be applied to TE1 at the first stop 310A to establishcalibrated target position (e.g. checkerboard position) represented bysix degrees of freedom (DOFs). TE1 at the first stop 310A is thereference for all the features (e.g. corners) on the target (e.g.checkerboard). Depending on how object points are constructed incalibrations or pose estimations, the origin of the estimated pose fromimages can be an upper right corner 302 of a checkerboard 300, as shownin FIG. 3B. 3D coordinates for all other features (e.g. corners) canthen be calculated with checker size per design and checkerboardpositions in six DOFs can be calibrated using a transformation matrix Tas follows:

$T = {\begin{pmatrix}{\cos\alpha\cos\beta} & {{\cos\alpha\sin\beta\sin\gamma} - {\sin\alpha\cos\gamma}} & {{\cos\alpha\sin\beta\cos\gamma} + {\sin\alpha\sin\gamma}} & x_{t} \\{\sin\alpha\cos\beta} & {{\sin{\alpha sin}\beta\sin\gamma} + {\cos\alpha\cos\gamma}} & {{\sin{\alpha sin}\beta\cos\gamma} - {\cos\alpha\sin\gamma}} & y_{t} \\{{- \sin}\beta} & {\cos\beta\sin\gamma} & {\cos\beta\cos\gamma} & z_{t} \\0 & 0 & 0 & 1\end{pmatrix}.}$

The transformation matrix includes six DOFs, which include threerotation angles α, ⊕, and γ and three translation coordinates x_(t),y_(t), and z_(t). The rotation sequence includes roll by γ, pitch by β,then yaw by α when the translations are zero. The rotation is betweenthe world coordinates and the camera coordinates. The translationcoordinates are also referred to as 3D coordinates.

With a plane normal to the camera (e.g. rotation angles are 0), 3Dcoordinates of all features (e.g. corners) can be calculated based onx_(t), y_(t), and z_(t) coordinates and checker size per design of thecheckerboard. Then, the transformation matrix can be applied to the 3Dcoordinates from a previous stop (e.g. first stop) to get the 3Dcoordinates for the current stop (e.g. second stop). The displacementsor distances between stops can be obtained from the 3D coordinates,x_(t), y_(t), and z_(t).

Once the 3D coordinates for all the features (e.g. corners) at the firststop are established, the 3D coordinates for all other stops can becalculated by applying relative transformations or displacementsmeasured from LiDAR sensors. Those 3D points can then be projected ontothe image plane by a camera matrix for predicted positions. The cameramatrix contains parameters such as focal length, principle points andscale factor. The camera matrix will be verified.

Note that extrinsic (R|t) is not needed since 3D points are representedby 3D coordinates in camera-frames. The discrepancy between predictedand detected positions is the indicator of camera calibration accuracy.

As shown in FIG. 3C, a target 308B is perpendicular to a linear stage204. Camera 302 can move from a first stop or initial position 310A tostops 310B-E. Displacement 1 is between the first stop 310A and thesecond stop 310B, which can be measured by LiDAR and can be estimatedbased upon the images of the target taken at these stops and cameracalibrations. Likewise, displacement 2 is between the first stop 310Aand the third stop 310C, which can also be measured by LiDAR sensors andcan be estimated based upon the images of the target taken at thesestops and camera calibrations.

In some variations, at least three targets are used for local lengthvalidation.

In some variations, at least five stops are made by the camera for locallength validation.

The focal length and image plane are defined in camera coordinates. FIG.4 illustrates camera coordinates X′, Y′, and Z′ with an origin locatedat a camera center and define a focal length from an image plane,according to some aspects of the disclosed technology. As shown, cameraframe 400A includes camera coordinates X′, Y′, and Z′ that have a cameracenter 402 at zero coordinates. An image plane 404 intersects with theZ′ axis at a focal length 406. The image plane 404 has 2D coordinatesalong u axis and v axis. The image plane 404 has an intersection withthe Z′ axis, which has 2D coordinates (u₀, v₀) and is referred to asprinciple point 408. A 2D point M is on the image plane 404.

Also, as shown in FIG. 4 , world frame 400B includes world coordinatesX, Y, and Z. The world coordinates X, Y, and Z can be converted to thecamera coordinates by a rotation matrix R and a translation matrix t,which combine to form a transformation matrix.

A calibration error is illustrated in the image plane. FIG. 5illustrates errors on the image plane of FIG. 4 , according to someaspects of the disclosed technology. As illustrated in FIG. 5 , afeature 508 is imaged by camera 502 on an image plane 504. Dot 506A isthe predicted position on the image plane, while dot 506B is thedetected position on the image plane. An error is a distance between dot506A and dot 506B.

Some images may be distorted. Some images may be undistorted. A fewexamples are provided below. FIG. 6A illustrates an example of imagedistortion, according to some aspects of the disclosed technology. Asillustrated in FIG. 6A, image 600A has a negative radial distortion.

FIG. 6B illustrates an example of undistorted image, according to someaspects of the disclosed technology. As illustrated in FIG. 6B, image600B has no distortion.

FIG. 6C illustrates another example of image distortion, according tosome aspects of the disclosed technology. As illustrated in FIG. 6C,image 600C has a positive radial distortion.

To have a linear relationship between focal lengths and displacements,the images from the camera need to be corrected for any distortions forlocal length validation. For the correction, a checkerboard may be usedas a target to get images. The checkerboard has straight lines that areparallel or perpendicular to each other. If there is any distortion, thestraight lines would become curved. In other words, a curved lineobserved from an image of the target indicates the presence ofdistortion. As such, the distortions can be corrected such that thecorrected image of the checkerboard with lines curved no more than 1pixel.

FIG. 7 illustrates an example method 700 for performing focal lengthvalidation, according to some aspects of the disclosed technology.Although example method 700 depicts a particular sequence of operations,the sequence may be altered without departing from the scope of thepresent disclosure. For example, some of the operations depicted may beperformed in parallel or in a different sequence that does notmaterially affect the function of method 700. In other examples,different components of an example device or system that implementsmethod 700 may perform functions at substantially the same time or in aspecific sequence.

According to some examples, method 700 includes receiving initial imagedata for an optical target from a camera at block 1010. For example, thecomputing system 1100 illustrated in FIG. 11 may receive initial imagedata for an optical target from a camera.

According to some examples, method 700 includes determining a distortionparameter associated with the camera, based on the initial image data atblock 1020. For example, the computing system 1100 illustrated in FIG.11 may determine a distortion parameter associated with the camera,based on the initial image data.

According to some examples, method 700 includes receiving a first imageof the optical target at block 730. For example, the computing system1100 illustrated in FIG. 11 may receive a first image of the opticaltarget. The first image is associated with a first position of thecamera.

According to some examples, method 700 includes receiving a second imageof the optical target at block 740. For example, the computing system1100 illustrated in FIG. 11 may receive a second image of the opticaltarget. The second image is associated with a second position of thecamera.

According to some examples, method 700 includes determining an estimateddisplacement between the first position of the camera and the secondposition of the camera based on the first image and the second image atblock 750. For example, the computing system 1100 illustrated in FIG. 11may determine an estimated displacement between the first position ofthe camera and the second position of the camera based on the firstimage and the second image.

According to some examples, method 700 includes determining a focallength based on the estimated displacement and a measured displacementof the camera between the first position and the second position atblock 760. For example, the computing system 1100 illustrated in FIG. 11may determine a focal length based on the estimated displacement and ameasured displacement of the camera between the first position and thesecond position.

According to some examples, method 700 includes receiving a third imageof the optical target. For example, the computing system 1100illustrated in FIG. 11 may receive a third image of the optical target.The third image is associated with a third position of the camera.

According to some examples, method 700 includes determining an estimateddisplacement between the first position of the camera and the thirdposition of the camera based on the first image and the second image.For example, the computing system 1100 illustrated in FIG. 11 maydetermine an estimated displacement between the first position of thecamera and the third position of the camera based on the first image andthe second image.

In some variations, the focal length is determined based on a ratio ofthe estimated displacement and the measured displacement.

In some variations, the measured displacement is determined using anoptical ranging device.

In some variations, the optical ranging device is a Leica scanner orLiDAR sensor.

In some variations, distortion corrections are performed on the firstimage and the second image based on the distortion parameter.

In some variations, the first target comprises a checkerboard.

EXAMPLES

The following examples are for illustration purposes only. It will beapparent to those skilled in the art that many modifications, both tomaterials and methods, may be practiced without departing from the scopeof the disclosure.

An example method for data collection may include step (1) checkingresidual distortion after correction with calibration results and makesure that the distortion coefficients meet the specification of linescurved no more than 1 pixel. The method may also include step (2)setting targets at a distance (e.g. 7 meters) away from the camera, andmaking sure that the three targets are in the middle of the verticalfield of views (FOVs), and with yaw at 0°, +45°, and −45°, respectively.

The method may also include step (3) catching target images using thecamera, measuring positions of each fiducial on the targets using LiDAR.

The method may also include step (4) moving the camera with a linearstage, catching target images with the camera including positions ofeach fiducial on the targets, and also measure positions of the cameraat each stop on the linear stage with LiDAR.

The method may also include repeating steps (3) to (4) five times.

The method may also include step (6) estimating target positions at eachstop using target images, calculating relative transformation betweenstops. For example, TE1 is a transformation to a first camera frame, TE2is a transformation to a second camera frame, then a relativetransformation TE_rel between a first stop of the camera and a secondstop of the camera 2 is TE1_inverse*TE2.

The method may also include step (7) correlating relativetransformations between stops from target images with LiDARmeasurements, and correct errors in image-based pose estimations.

The method may also include step (8) applying corrections to poseestimations with the image at the first stop to extract targetpositions, which can then be used in combination with relativetransformations TLi for target positions in camera-frames for all thestops of the camera.

The method may also include step (9) calculating predicted positions byprojecting 3D points onto the image plane using the camera matrix to beevaluated, and compare the predicted positions with detected imagepoints. Step (9) is repeated for each target at each stop.

FIG. 8 illustrates displacements from pose estimations with images andmeasured displacements from LiDAR, according to some aspects of thedisclosed technology. As illustrated in FIG. 8 , the displacements frompose estimation with images are linearly correlated with thedisplacements from LiDAR. The slope of line 802 is 1.0064 and theintercept is −0.001.

FIG. 9 illustrates simulated results of target distances versus focallengths, according to some aspects of the disclosed technology. Asillustrated in FIG. 9 , target distance is linearly correlated with thefocal length as illustrated by dots 902.

The inaccuracy in estimated positions from images may be due tocalibration errors. The inaccuracy in estimated positions from imagesmay be linear with calibration errors, if the images are not distorted.As such, the discrepancy between the estimated positions from images andthe measured positions from LiDAR correlate well with the calibrationerrors.

Orientation measurement errors may occur in LiDAR measurements. WhenLiDAR has an angle resolution of 0.0012 rad, an error of 0.01 pixel iscreated in the image plane. For a range resolution of 100 μm for LiDAR,fiducials need to be at least 200 mm away from each other with 2×margin.

Range measurement errors may occur in LiDAR measurements due to itslimited resolution. As shown in FIG. 8 , distances correlate with focallengths linearly. To detect a 0.1% variation in the focal length,displacement measurements by LiDAR need to have an accuracy of 0.1%,which translates to 100 μm with a 1 m distance between each stop. Thisis consistent with the LiDAR resolution.

Orientation errors occur from image-based pose estimations. Both theorientation errors in image-based pose estimations and calibrationinaccuracy can result in the orientation errors in estimated targetorientations, which further cause inaccuracy in coordinates of the 3Dpoints (e.g. coordinates of the corners).

In some aspects, the focal length may be validated automatically on anautonomous vehicle (AV). The AV includes camera and Lidar sensors. Whenthe AV drives linearly, the camera on the AV moves like a linear stage.

FIG. 10 illustrates environment 1000 that includes an autonomous vehicle1002 in communication with a computing system 750. The autonomousvehicle 1002 can navigate about roadways without a human driver basedupon sensor signals output by sensor systems 1004-1006 of the autonomousvehicle 1002. The autonomous vehicle 1002 includes a plurality of sensorsystems 1004-1006 (a first sensor system 1002 through an Nth sensorsystem 1004). The sensor systems 1004-1006 are of different types andare arranged about the autonomous vehicle 1002. For example, the firstsensor system 1004 may be a camera sensor system, and the Nth sensorsystem 1006 may be a lidar sensor system. Other exemplary sensor systemsinclude radar sensor systems, global positioning system (GPS) sensorsystems, inertial measurement units (IMU), infrared sensor systems,laser sensor systems, sonar sensor systems, and the like.

The autonomous vehicle 1002 further includes several mechanical systemsthat are used to effectuate the appropriate motion of the autonomousvehicle 1002. For instance, the mechanical systems can include but arenot limited to, a vehicle propulsion system 730, a braking system 732,and a steering system 734. The vehicle propulsion system 730 may includean electric motor, an internal combustion engine, or both. The brakingsystem 732 can include an engine brake, brake pads, actuators, and/orany other suitable componentry that is configured to assist indecelerating the autonomous vehicle 1002. The steering system 734includes suitable componentry that is configured to control thedirection of movement of the autonomous vehicle 1002 during navigation.

The autonomous vehicle 1002 further includes a safety system 736 thatcan include various lights and signal indicators, parking brake,airbags, etc. The autonomous vehicle 1002 further includes a cabinsystem 738 that can include cabin temperature control systems, in-cabinentertainment systems, etc.

The autonomous vehicle 1002 additionally comprises an internal computingsystem 1010 that is in communication with the sensor systems 1004-1006and the mechanical systems 730, 732, 734. The internal computing systemincludes at least one processor and at least one memory havingcomputer-executable instructions that are executed by the processor. Thecomputer-executable instructions can make up one or more servicesresponsible for controlling the autonomous vehicle 1002, communicatingwith remote computing system 750, receiving inputs from passengers orhuman co-pilots, logging metrics regarding data collected by sensorsystems 1004-1006 and human co-pilots, etc.

The internal computing system 1010 can include a control service 1012that is configured to control operation of the vehicle propulsion system206, the braking system 208, the steering system 210, the safety system736, and the cabin system 738. The control service 1012 receives sensorsignals from the sensor systems 202-204 as well communicates with otherservices of the internal computing system 1010 to effectuate operationof the autonomous vehicle 1002. In some embodiments, control service1012 may carry out operations in concert one or more other systems ofautonomous vehicle 1002.

The control service 1012 may include the control system 204 asillustrated in FIG. 2 to communicate with the sensors including airbagsensors or accelerators. The control system 204 may also communicatewith the release mechanism 206 via high current relay switch 306.

The internal computing system 1010 can also include a constraint service1014 to facilitate safe propulsion of the autonomous vehicle 1002. Theconstraint service 1016 includes instructions for activating aconstraint based on a rule-based restriction upon operation of theautonomous vehicle 1002. For example, the constraint may be arestriction upon navigation that is activated in accordance withprotocols configured to avoid occupying the same space as other objects,abide by traffic laws, circumvent avoidance areas, etc. In someembodiments, the constraint service can be part of the control service1012

The internal computing system 1010 can also include a communicationservice 1016. The communication service can include both software andhardware elements for transmitting and receiving signals from/to theremote computing system 250. The communication service 1016 isconfigured to transmit information wirelessly over a network, forexample, through an antenna array that provides personal cellular(long-term evolution (LTE), 3G, 5G, etc.) communication.

In some embodiments, one or more services of the internal computingsystem 1010 are configured to send and receive communications to remotecomputing system 750 for such reasons as reporting data for training andevaluating machine learning algorithms, requesting assistance fromremoting computing system or a human operator via remote computingsystem, software service updates, ridesharing pickup and drop offinstructions etc.

The internal computing system 1010 can also include a latency service1018. The latency service 1018 can utilize timestamps on communicationsto and from the remote computing system 750 to determine if acommunication has been received from the remote computing system 750 intime to be useful. For example, when a service of the internal computingsystem 1010 requests feedback from remote computing system 750 on atime-sensitive process, the latency service 1018 can determine if aresponse was timely received from remote computing system 750 asinformation can quickly become too stale to be actionable. When thelatency service 1018 determines that a response has not been receivedwithin a threshold, the latency service 1018 can enable other systems ofautonomous vehicle 1002 or a passenger to make necessary decisions or toprovide the needed feedback.

The internal computing system 1010 can also include a user interfaceservice 120 that can communicate with cabin system 738 in order toprovide information or receive information to a human co-pilot or humanpassenger. In some embodiments, a human co-pilot or human passenger maybe required to evaluate and override a constraint from constraintservice 1014, or the human co-pilot or human passenger may wish toprovide an instruction to the autonomous vehicle 1002 regardingdestinations, requested routes, or other requested operations.

As described above, the remote computing system 750 is configured tosend/receive a signal from the autonomous vehicle regarding reportingdata for training and evaluating machine learning algorithms, requestingassistance from remoting computing system or a human operator via theremote computing system 750, software service updates, ridesharingpickup and drop off instructions, etc.

The remote computing system 750 includes an analysis service 752 that isconfigured to receive data from autonomous vehicle 1002 and analyze thedata to train or evaluate machine learning algorithms for operating theautonomous vehicle 1002. The analysis service 752 can also performanalysis pertaining to data associated with one or more errors orconstraints reported by autonomous vehicle 1002.

The remote computing system 750 can also include a user interfaceservice 754 configured to present metrics, video, pictures, soundsreported from the autonomous vehicle 1002 to an operator of remotecomputing system 750. User interface service 754 can further receiveinput instructions from an operator that can be sent to the autonomousvehicle 1002.

The remote computing system 750 can also include an instruction service756 for sending instructions regarding the operation of the autonomousvehicle 1002. For example, in response to an output of the analysisservice 752 or user interface service 754, instructions service 756 canprepare instructions to one or more services of the autonomous vehicle1002 or a co-pilot or passenger of the autonomous vehicle 1002.

The remote computing system 750 can also include a rideshare service 758configured to interact with ridesharing applications operating on(potential) passenger computing devices. The rideshare service 758 canreceive requests to be picked up or dropped off from passengerridesharing app 770 and can dispatch autonomous vehicle 1002 for thetrip. The rideshare service 758 can also act as an intermediary betweenthe ridesharing app 770 and the autonomous vehicle wherein a passengermight provide instructions to the autonomous vehicle to 1002 go aroundan obstacle, change routes, honk the horn, etc.

FIG. 11 shows an example of computing system 1100, which can be, forexample, used for all the calculations as discussed above, or can be anycomputing device making up internal computing system 1010, remotecomputing system 1050, (potential) passenger device executing rideshareapp 1070, or any component thereof in which the components of the systemare in communication with each other using connection 1105. Connection1105 can be a physical connection via a bus, or a direct connection intoprocessor 1110, such as in a chipset architecture. Connection 1105 canalso be a virtual connection, networked connection, or logicalconnection.

In some embodiments, computing system 1100 is a distributed system inwhich the functions described in this disclosure can be distributedwithin a datacenter, multiple data centers, a peer network, etc. In someembodiments, one or more of the described system components representsmany such components each performing some or all of the function forwhich the component is described. In some embodiments, the componentscan be physical or virtual devices.

Example system 1100 includes at least one processing unit (CPU orprocessor) 1110 and connection 1105 that couples various systemcomponents including system memory 1115, such as read-only memory (ROM)1120 and random access memory (RAM) 1125 to processor 1110. Computingsystem 1100 can include a cache of high-speed memory 1112 connecteddirectly with, in close proximity to, or integrated as part of processor1110.

Processor 1110 can include any general purpose processor and a hardwareservice or software service, such as services 1132, 1134, and 1136stored in storage device 1130, configured to control processor 1110 aswell as a special-purpose processor where software instructions areincorporated into the actual processor design. Processor 1110 mayessentially be a completely self-contained computing system, containingmultiple cores or processors, a bus, memory controller, cache, etc. Amulti-core processor may be symmetric or asymmetric.

To enable user interaction, computing system 1100 includes an inputdevice 1145, which can represent any number of input mechanisms, such asa microphone for speech, a touch-sensitive screen for gesture orgraphical input, keyboard, mouse, motion input, speech, etc. Computingsystem 1100 can also include output device 1135, which can be one ormore of a number of output mechanisms known to those of skill in theart. In some instances, multimodal systems can enable a user to providemultiple types of input/output to communicate with computing system1100. Computing system 1100 can include communications interface 1140,which can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement, and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1130 can be a non-volatile memory device and can be ahard disk or other types of computer readable media which can store datathat are accessible by a computer, such as magnetic cassettes, flashmemory cards, solid state memory devices, digital versatile disks,cartridges, random access memories (RAMs), read-only memory (ROM),and/or some combination of these devices.

The storage device 1130 can include software services, servers,services, etc., that when the code that defines such software isexecuted by the processor 1110, it causes the system to perform afunction. In some embodiments, a hardware service that performs aparticular function can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as processor 1110, connection 1105, output device 1135,etc., to carry out the function.

For clarity of explanation, in some instances, the present technologymay be presented as including individual functional blocks includingfunctional blocks comprising devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Any of the steps, operations, functions, or processes described hereinmay be performed or implemented by a combination of hardware andsoftware services or services, alone or in combination with otherdevices. In some embodiments, a service can be software that resides inmemory of a client device and/or one or more servers of a contentmanagement system and perform one or more functions when a processorexecutes the software associated with the service. In some embodiments,a service is a program or a collection of programs that carry out aspecific function. In some embodiments, a service can be considered aserver. The memory can be a non-transitory computer-readable medium.

In some embodiments, the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer-readable media. Such instructions can comprise,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The executable computer instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, solid-state memory devices, flash memory, USB devices providedwith non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprisehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include servers,laptops, smartphones, small form factor personal computers, personaldigital assistants, and so on. The functionality described herein alsocan be embodied in peripherals or add-in cards. Such functionality canalso be implemented on a circuit board among different chips ordifferent processes executing in a single device, by way of furtherexample.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

What is claimed:
 1. A computer-implemented method comprising: receiving,by a computing system, a first image of an optical target from a camera,wherein the first image is associated with a first position of thecamera; receiving, by the computing system, a second image of theoptical target, wherein the second image is associated with a secondposition of the camera; determining an estimated displacement betweenthe first position of the camera and the second position of the camerabased on the first image and the second image; and determining a focallength based on the estimated displacement and a measured displacementof the camera between the first position and the second position.
 2. Thecomputer-implemented method of claim 1, further comprising: receiving,by the computing system, image data for the optical target from thecamera; and determining a distortion parameter associated with thecamera, based on the image data, wherein distortion corrections areperformed on the first image and the second image based on thedistortion parameter.
 3. The computer-implemented method of claim 1,wherein the focal length is based on a ratio of the estimateddisplacement and the measured displacement.
 4. The computer-implementedmethod of claim 1, wherein the measured displacement is determined usingan optical ranging device.
 5. The computer-implemented method of claim3, wherein the optical ranging device is a Light Detection and Ranging(LiDAR) sensor.
 6. The computer-implemented method of claim 1, whereinthe first target comprises a checkerboard.
 7. The computer-implementedmethod of claim 1, further comprising: receiving, by the computingsystem, a third image of the optical target, wherein the third image isassociated with a third position of the camera; and determining anestimated displacement between the first position of the camera and thethird position of the camera based on the first image and the secondimage.
 8. A system comprising: a storage device configured to storeinstructions; a processor configured to execute the instructions andcause the processor to: receive a first image of an optical target froma camera, wherein the first image is associated with a first position ofthe camera, receive a second image of the optical target wherein thesecond image is associated with a second position of the camera,determine an estimated displacement between the first position of thecamera and the second position of the camera based on the first imageand the second image, and determine a focal length based on theestimated displacement and a measured displacement of the camera betweenthe first position and the second position.
 9. The system of claim 8,wherein the processor is configured to execute the instructions andcause the processor to: receive image data for the optical target fromthe camera; and determine a distortion parameter associated with thecamera, based on the image data, wherein distortion corrections areperformed on the first image and the second image based on thedistortion parameter.
 10. The system of claim 8, wherein the focallength is based on a ratio of the estimated displacement and themeasured displacement.
 11. The system of claim 8, wherein the measureddisplacement is determined using an optical ranging device.
 12. Thesystem of claim 10, wherein the optical ranging device is a Leica.scanner or LiDAR sensor.
 13. The system of claim 8, wherein the firsttarget comprises a checkerboard.
 14. The system of claim 8, wherein theprocessor is configured to execute the instructions and cause theprocessor to: receive a third image of the optical target, wherein thethird image is associated with a third position of the camera; anddetermine an estimated displacement between the first position of thecamera and the third position of the camera based on the first image andthe second image
 15. A non-transitory computer readable mediumcomprising instructions, the instructions, when executed by a computingsystem, cause the computing system to: receive a first image of anoptical target from a camera, wherein the first image is associated witha first position of the camera; receive a second image of the opticaltarget, wherein the second image is associated with a second position ofthe camera; determine an estimated displacement between the firstposition of the camera and the second position of the camera based onthe first image and the second image; and determine a focal length basedon the estimated displacement and a measured displacement of the camerabetween the first position and the second position.
 16. The computerreadable medium of claim 15, wherein the computer readable-mediumfurther comprises the instructions that, when executed by the computingsystem, cause the computing system to: receive image data for theoptical target from the camera; and determine a distortion parameterassociated with the camera, based on the image data, wherein distortioncorrections are performed on the first image and the second image basedon the distortion parameter.
 17. The computer readable medium of claim15, wherein the focal length is based on a ratio of the estimateddisplacement and the measured displacement.
 18. The computer readablemedium of claim 15, wherein the measured displacement is determinedusing an optical ranging device.
 19. The computer readable medium ofclaim 17, wherein the optical ranging device is a Leica scanner or LiDARsensor.
 20. The computer readable medium of claim 15, wherein the firsttarget comprises a checkerboard.